Reaching for the Sky: Maximizing Deep Learning Inference Throughput on Edge Devices with AI Multi-Tenancy

The wide adoption of smart devices and Internet-of-Things (IoT) sensors has led to massive growth in data generation at the edge Internet over past decade. Intelligent real-time analysis such a high volume data, particularly leveraging highly accurate deep learning (DL) models, often requires be processed as close sources (or Internet) minimize network processing latency. advent specialized, low-cost, power-efficient greatly facilitated DL inference tasks edge. However, limited research been done improve throughput (e.g., number inferences per second) by exploiting various system techniques. This study investigates techniques, batched inferencing, AI multi-tenancy, cluster accelerators, which can significantly enhance overall on with models for image classification tasks. In particular, multi-tenancy enables collective utilization devices’ resources (CPU, GPU) accelerators Edge Tensor Processing Units; EdgeTPUs). evaluation results show that inferencing more than 2.4× improvement equipped high-performance GPUs like Jetson Xavier NX. Moreover, approaches, e.g., concurrent model executions (CME) dynamic placements (DMP), (with GPUs) EdgeTPU further improved up 3× 10×, respectively. Furthermore, we present detailed hardware software factors change EdgeTPUs, thereby shedding light areas could achieve